In recent years, development of an AR (Augmented Reality) technique has been advanced in which a virtual object is superimposed onto an image of a real world and the resultant image is displayed, whereby the virtual object appears to be present in the real world.
For example, a stereoscopic display apparatus disclosed in Patent Literature 1 (Japanese Laid-Open Patent Publication No. 2008-146109) calculates the relative positions and relative orientations of a camera for a right eye and a camera for a left eye with respect to a marker placed in the real world, from respective images shot by the camera for a right eye and the camera for a left eye, which are attached to a head mounted display, and then, based on the result of the calculation, generates an image of a virtual object for a right eye and an image for the virtual object for a left eye. Then, the stereoscopic display apparatus superimposes the image of the virtual object for a right eye and the image for the virtual object for a left eye onto the images shot by the camera for a right eye and the camera for a left eye, respectively, and displays the resultant composite images on an LCD (liquid crystal display) for a right eye and an LCD for a left eye, respectively.
In addition, Non-Patent Literature 1 (Hirokazu Kato, Mark Billinghurst, “Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System,” iwar, pp. 85, 2nd IEEE and ACM International Workshop on Augmented Reality, 1999) discloses a method of, based on the position and the orientation of a marker in an image shot by a camera, calculating the relative positions and the relative orientations of the marker and the camera with respect to each other in the real world.
However, in the case where the relative positions and the relative orientations of the marker and the camera with respect to each other in the real world are calculated based on the position and the orientation of the marker in an image shot by the camera, if the image shot by the camera is blurred or if the accuracy of marker recognition is not perfect, error occurs in a result of calculation. Therefore, in the stereoscopic display apparatus disclosed in Patent Literature 1, the relative position and orientation of the camera for a right eye with respect to the marker calculated based on an image shot by the camera for a right eye are not always accurate, and also, the relative position and orientation of the camera for a left eye with respect to the marker calculated based on an image shot by the camera for a left eye are not always accurate. An image for a right eye including a virtual object, and an image for a left eye including the virtual object that are generated based on a result of calculation that is not accurate, do not have consistency with each other, that is, they are inconsistence with each other. As a result, a user cannot normally stereoscopically view the virtual object.
Therefore, an object of the present invention is to provide a computer-readable storage medium, an image display apparatus, an image display system, and an image display method that are capable of displaying a virtual object so as to be normally stereoscopically visible.
The present invention has the following features to attain the object mentioned above.
A computer-readable storage medium of the present invention is a computer-readable storage medium having stored therein an image display program which stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, the image display program causing a computer to function as first real world image obtaining means, second real world image obtaining means, camera relative position relationship calculation means, virtual camera setting means, first virtual space image generation means, second virtual space image generation means, and display control means. The first real world image obtaining means obtains a first real world image shot by a real camera for a left eye. The second real world image obtaining means obtains a second real world image shot by a real camera for a right eye. The camera relative position relationship calculation means calculates a relative position relationship between the real camera for a left eye and the real camera for a right eye, based on a manner in which a predetermined shooting target is visible on the first real world image and a manner in which the predetermined shooting target is visible on the second real world image. The virtual camera setting means determines the position of a left virtual camera for generating an image for a left eye, and the position of a right virtual camera for generating an image for a right eye, in a predetermined virtual space, so as to be separated from each other in accordance with the relative position relationship calculated by the camera relative position relationship calculation means. The first virtual space image generation means generates a first virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera. The second virtual space image generation means generates a second virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera. The display control means superimposes the first virtual space image onto the first real world image, and superimposes the second virtual space image onto the second real world image, to output images for a stereoscopic view to the stereoscopic display apparatus.
According to the above configuration, the relative position relationship between the two virtual cameras are calculated by using the real world images shot by the two real cameras, and the positions of the two virtual cameras in the virtual space are determined based on the relative position relationship which has been calculated. Therefore, for example, even if the relative position relationship between the two real cameras is not known, or even if there is error in the attachment of the two real camera, a relative position relationship between the right virtual camera and the left virtual camera can be set appropriately, and a virtual object can appropriately be displayed in a stereoscopic manner.
In another preferable configuration, the image display program may further cause the computer to function as: first position orientation calculation means for calculating at least relative positions of the real camera for a left eye and the predetermined shooting target with respect to each other, based on the position and the orientation of the predetermined shooting target in the first real world image; and second position orientation calculation means for calculating at least relative positions of the real camera for a right eye and the predetermined shooting target with respect to each other, based on the position and the orientation of the predetermined shooting target in the second real world image. The camera relative position relationship calculation means may calculate the relative position relationship between the real camera for a left eye and the real camera for a right eye, based on the relative position of the real camera for a left eye with respect to the predetermined shooting target which is calculated by the first position orientation calculation means, and on the relative position of the real camera for a right eye with respect to the predetermined shooting target which is calculated by the second position orientation calculation means.
In another preferable configuration, at least one of the first position orientation calculation means and the second position orientation calculation means may calculate relative orientations of the corresponding one of the real cameras and the predetermined shooting target with respect to each other, in addition to the relative positions, based on the position and the orientation of the predetermined shooting target in the corresponding one of the real world images. The virtual camera setting means may determine the orientations of the left virtual camera and the right virtual camera, based on the relative orientations of the real camera and the predetermined shooting target which are calculated by the at least one of the first position orientation calculation means and the second position orientation calculation means.
In another preferable configuration, the relative position relationship which is calculated by the camera relative position relationship calculation means may be the interval between the real camera for a left eye and the real camera for a right eye.
In another preferable configuration, the camera relative position relationship calculation means calculates the relative position relationship between the real camera for a left eye and the real camera for a right eye, based on the difference between the relative position of the predetermined shooting target with respect to the real camera for a left eye which is calculated based on the first real world image, and the relative position of the predetermined shooting target with respect to the real camera for a right eye which is calculated based on the second real world image.
In another preferable configuration, the image display program may further cause the computer to function as: first transform matrix calculation means for, based on the first real world image, calculating a first transform matrix that transforms coordinates represented in a coordinate system based on the predetermined shooting target, into coordinates represented in a coordinate system based on the real camera for a left eye; and second transform matrix calculation means for, based on the second real world image, calculating a second transform matrix that transforms coordinates represented in the coordinate system based on the predetermined shooting target, into coordinates represented in a coordinate system based on the real camera for a right eye. The camera relative position relationship calculation means may multiply a 3-dimensional vector indicating an origin by one of the first transform matrix and the second transform matrix, and multiply the resultant vector by the inverse matrix of the other one, to calculate a relative position vector indicating the relative position relationship between the real camera for a left eye and the real camera for a right eye. The virtual camera setting means may set the interval between the left virtual camera and the right virtual camera, based on the magnitude of the relative position vector.
In another preferable configuration, the virtual camera setting means may set the orientations of the left virtual camera and the right virtual camera such that the shooting directions thereof are in parallel with each other.
In another preferable configuration, the image display program may further cause the computer to function as scale relevant processing means for executing, in the virtual space, predetermined processing in accordance with a scale in the real world, based on the relative position relationship calculated by the camera relative position relationship calculation means, and on a designed relative position relationship between the real camera for a left eye and the real camera for a right eye.
According to the above configuration, it is possible to execute processing in accordance with a scale in the real world, in the virtual space.
In another preferable configuration, the image display program may further cause the computer to function as depth variation determination means for determining whether or not the depth distance from the real camera for a left eye or the real camera for a right eye to the predetermined shooting target has varied beyond a predetermined range in comparison with the depth distance as it was when the camera relative position relationship calculation means calculated the relative position relationship between the real camera for a left eye and the real camera for a right eye. The camera relative position relationship calculation means may keep the relative position relationship that was last determined, without calculating again the relative position relationship between the real camera for a left eye and the real camera for a right eye, as long as the result of the determination by the depth variation determination means is negative.
According to the above configuration, the relative position relationship between the real camera for a left eye and the real camera for a right eye does not need to be constantly updated, and as a result, a load of processing can be reduced.
In another preferable configuration, the image display program may further cause the computer to function as depth variation determination means for determining whether or not the depth distance from the real camera for a left eye or the real camera for a right eye to the predetermined shooting target has varied beyond a predetermined range in comparison with the depth distance as it was when the camera relative position relationship calculation means calculated the relative position relationship between the real camera for a left eye and the real camera for a right eye. The camera relative position relationship calculation means may calculate again the relative position relationship between the real camera for a left eye and the real camera for a right eye, when the result of the determination by the depth variation determination means is positive.
According to the above configuration, reliability of the calculated relative position relationship between the real camera for a left eye and the real camera for a right eye can be prevented from being reduced.
In another preferable configuration, the image display program may further cause the computer to function as depth distance calculation means for calculating the depth distance from the real camera for a left eye or the real camera for a right eye to the predetermined shooting target, based on the real world image including the predetermined shooting target. The depth variation determination means may determine whether or not the depth distance from the real camera for a left eye or the real camera for a right eye to the predetermined shooting target has varied beyond the predetermined range by comparing a reference depth distance calculated by the depth distance calculation means at a time when the camera relative position relationship calculation means calculated the relative position relationship between the real camera for a left eye and the real camera for a right eye, with the latest depth distance calculated by the depth distance calculation means after the calculation of the reference depth distance.
In another preferable configuration, the depth distance calculation means may execute processing of calculating the depth distance from the real camera for a left eye or the real camera for a right eye to the predetermined shooting target every time a new real world image is obtained by the first real world image obtaining means or the second real world image obtaining means, and may calculate the reference depth distance, based on results of a plurality of the calculations of the depth distance obtained by the processing being executed a plurality of times.
According to the above configuration, reliability of the calculated reference depth distance is improved.
In another preferable configuration, the depth distance calculation means may determine the reference depth distance, based on the results of the plurality of calculations of the depth distance, only when all the results of the plurality of calculations of the depth distance are within a predetermined range.
According to the above configuration, reliability of the calculated reference depth distance is improved.
An image display apparatus according to the present invention stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, and comprises first real world image obtaining means, second real world image obtaining means, camera relative position relationship calculation means, virtual camera setting means, first virtual space image generation means, second virtual space image generation means, and display control means. The first real world image obtaining means obtains a first real world image shot by a real camera for a left eye. The second real world image obtaining means obtains a second real world image shot by a real camera for a right eye. The camera relative position relationship calculation means calculates a relative position relationship between the real camera for a left eye and the real camera for a right eye, based on a manner in which a predetermined shooting target is visible on the first real world image and a manner in which the predetermined shooting target is visible on the second real world image. The virtual camera setting means determines the position of a left virtual camera for generating an image for a left eye, and the position of a right virtual camera for generating an image for a right eye, in a predetermined virtual space, so as to be separated from each other in accordance with the relative position relationship calculated by the camera relative position relationship calculation means. The first virtual space image generation means generates a first virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera. The second virtual space image generation means generates a second virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera. The display control means superimposes the first virtual space image onto the first real world image, and superimposes the second virtual space image onto the second real world image, to output images for a stereoscopic view to the stereoscopic display apparatus.
An image display system according to the present invention stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, and comprises a predetermined shooting target, first real world image obtaining means, second real world image obtaining means, camera relative position relationship calculation means, virtual camera setting means, first virtual space image generation means, second virtual space image generation means, and display control means. The first real world image obtaining means obtains a first real world image shot by a real camera for a left eye. The second real world image obtaining means obtains a second real world image shot by a real camera for a right eye. The camera relative position relationship calculation means calculates a relative position relationship between the real camera for a left eye and the real camera for a right eye, based on a manner in which the predetermined shooting target is visible on the first real world image and a manner in which the predetermined shooting target is visible on the second real world image. The virtual camera setting means determines the position of a left virtual camera for generating an image for a left eye, and the position of a right virtual camera for generating an image for a right eye, in a predetermined virtual space, so as to be separated from each other in accordance with the relative position relationship calculated by the camera relative position relationship calculation means. The first virtual space image generation means generates a first virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera. The second virtual space image generation means generates a second virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera. The display control means superimposes the first virtual space image onto the first real world image, and superimposes the second virtual space image onto the second real world image, to output images for a stereoscopic view to the stereoscopic display apparatus.
An image display method according to the present invention stereoscopically displays a real world image onto which a 3-dimensional virtual object is superimposed, on a screen of a stereoscopic display apparatus capable of providing a stereoscopic view, and comprises a first real world image obtaining step, a second real world image obtaining step, a camera relative position relationship calculation step, a virtual camera setting step, a first virtual space image generation step, a second virtual space image generation step, and a display control step. The first real world image obtaining step obtains a first real world image shot by a real camera for a left eye. The second real world image obtaining step obtains a second real world image shot by a real camera for a right eye. The camera relative position relationship calculation step calculates a relative position relationship between the real camera for a left eye and the real camera for a right eye, based on a manner in which a predetermined shooting target is visible on the first real world image and a manner in which the predetermined shooting target is visible on the second real world image. The virtual camera setting step determines the position of a left virtual camera for generating an image for a left eye, and the position of a right virtual camera for generating an image for a right eye, in a predetermined virtual space, so as to be separated from each other in accordance with the relative position relationship calculated in the camera relative position relationship calculation step. The first virtual space image generation step generates a first virtual space image indicating the predetermined virtual space as it is looked at from the left virtual camera. The second virtual space image generation step generates a second virtual space image indicating the predetermined virtual space as it is looked at from the right virtual camera. The display control step superimposes the first virtual space image onto the first real world image, and superimposes the second virtual space image onto the second real world image, to output images for a stereoscopic view to the stereoscopic display apparatus.
According to the present invention, it is possible to display a virtual object so as to be normally stereoscopically visible.
These and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.